Statistical Signal Processing
This page contains resources about Statistical Signal Processing, Point Estimation, Estimation Theory, Adaptive Filtering, Adaptive Signal Processing and Adaptive Filter Theory, System Identification and Adaptive Array Processing. Subfields and Concepts Signal Modelling & Spectral Estimation See also Digital Signal Processing and Stochastic Processes * Signal Modelling ** Linear Nonparametric Signal Models *** Linear random signal model / General linear model *** Recursive representation *** Innovations representation ** Parametric Pole-Zero Signal Models / Time Series Models *** Autoregressive (AR) model / All-Pole model *** Moving Average (MA) model / All-Zero model *** ARMA model / Pole-Zero model ** Linear Predictive Coding *** Reflection Coefficients *** Partial Correlation Coefficients ** Least Squares Method ** Padé Approximation ** Levinson–Durbin Algorithm / Levinson–Durbin Recursion ** Prony's Method ** Itakura–Saito / Lattice filter ** Maximum Entropy Method (MEM) ** Iterative Prefiltering ** Final Data Records / Block Estimation / Data Windowing / Finite Interval Modelling *** Yule Walker Method / Autocorrelation Method *** Covariance Method *** Modified Covariance *** Pre-windowing Method *** Post-windowing Method *** Unbiased Autocorrelation Estimate *** Burg Method *** Forward and Backward Linear Prediction (FBLP) ** Stochastic Modelling *** Modified Yule-Walker Equation (MYWE) Method *** Least Squares MYWE Method *** MA Model using Spectral Factorization *** Durbin's Method * Stochastic Systems ** Stochastic Processes (e.g. Wiener Process) ** Stochastic Calculus ** Stochastic Optimal Control ** Stochastic Time Series * (Nonparametric) Time-domain Methods ** Autocorrelation Function ** Cross-Correlation ** Cross-Covariance ** Correlogram * Frequency-domain Methods / Spectral Estimation ** Nonparametric Methods *** Periodogram *** Modified Periodogram *** Barlett's Method / Periodogram Averaging *** Welch's Method / Modified Periodogram Averaging *** Blackman-Tukey Method / Periodogram Smoothing *** Multiple Window Method *** Cross-spectrum / Magnitude squared coherence *** Empirical Transfer Function Estimation (ETFE) *** Minimum Variance Spectral Estimation (MVSE) / Capon's Method ** Maximum Entropy Method (MEM) ** Parametric Methods *** AR Spectral Estimation (using All-Pole Signal Modelling) *** MA Spectral Estimation (using All-Zero Signal Modelling) *** ARMA Spectral Estimation (using Pole-Zero Signal Modelling) *** Minimum Variance Distortionless Response (MVDR) filter *** Linearly Constrained Minimum Variance (LCMV) filter *** Subspace Methods ** Frequency Estimation (using a Harmonic Process / Sinusoid Model) *** Pisarenko Harmonic Decomposition *** Multiple Signal Classification (MUSIC) pseudospectrum *** Eigenvector Method *** Root-MUSIC pseudospectrum *** Minimum Norm Method *** ESPRIT Algorithm *** Pencil Method *** Frequency-domain version of Prony's Method *** Subspace Methods / Eigendecomposition-based Methods **** Blackman-Tukey Principal Components Method **** AR Method **** Minimum Variance Method ** High-resolution / Super-resolution Spectral Estimators *** MVSE *** MVDR *** LCMV *** MUSIC Adaptive Filtering & Optimal Filtering See also Optimization * Linear Shift-Invariant (LSI) filter / LTI filter * Filtering Problem * Adaptive filter ** Shift-Varying filter structure ** Criterion of performance ** Adaptive Algorithm * Four major configurations of an adaptive filter ** System Identification ** Noise Cancellation ** Prediction ** Inverse System Identification * Basic filter representations ** Transversal (direct form) filter structure ** Symmetric transversal filter structure ** Lattice filter structure ** Parallel form filter structure ** State-space representation ** Innovations representation / Innovations filter structure * FIR filter / Non-recursive adaptive filter ** Gradient Descent / Steepest Descent ** Normalized Nonlinear Gradient Descent (NNGD) ** Fully Adaptive NNGD (FANNGD) ** Unnormalized Gradient / Least Mean Squares (LMS) filter ** Normalized Gradient / Normalized LMS (NLMS) filter ** Leaky LMS filter ** Gradient Adaptive Lattice (GAL) filter ** Lattice LMS filter with Joint Process Estimation ** FIR Wiener filter * IIR filter / Recursive adaptive filter ** Recursive Least Squares (RLS) filter / Forgetting Factor ** Kernel RLS (KRLS) filter ** Sliding Window RLS filter ** IIR Wiener filter * Polynomial Regression filters * Gaussian Regression filter * Online State Estimation / Recursive Estimation ** Bayesian Recursive Estimation / Bayes filter ** Kalman filter ** Extended Kalman filter (EKF) ** Unscented Kalman filter (UKF) ** Iterated EKF ** Information filter ** Interacting Multiple Models (IMM) Filter ** Histogram filter ** Monte Carlo Methods (Approximation to Bayesian Estimation) *** Particle filter * Optimum filters ** Eigenfilter ** Kalman filter ** Wiener filter ** Linear Prediction *** Forward Linear Prediction *** Backward Linear Prediction * Square Root filter / Cholesky Decomposition-based Kalman Filter ** Exponentially Weighted RLS filter ** QR Decomposition-based RLS (QRD-RLS) ** Extended QRD-RLS ** Inverse QRD-RLS * Hierarchical filters ** Hierarchical Gradient Descent (HGD) ** Hierarchical LMS (HLMS) filter * Complex valued filters ** Complex LMS (CLMS) filter * Iterative Methods in Optimization ** Expectation-Maximization (EM) Algorithm ** Gradient Descent / Steepest Descent ** Levenberg–Marquardt Algorithm ** Iteratively Reweighted Least Squares ** Nonlinear Least Squares ** Krylov Subspace Methods *** Conjugate Gradient Method ** Broyden–Fletcher–Goldfarb–Shanno (BFGS) Algorithm * Risk Function / Expected Loss (i.e. Expectation Value of Loss Function) ** Mean Square Error (MSE) * Criteria of performance ** Mean squared error (MSE) / Minimum MSE estimator ** Mean n-th order error ** Sum of squared errors (SSE) ** Signal-to-Noise Ratio (SNR) * Stochastic Optimization ** Stochastic Approximation * Adaptive Control ** Auto-tuning / Self-tuning * Adaptive Array Processing ** Adaptive beamforming ** Matched subspace filter / Matched subspace detector ** Angle estimation ** Space-time adaptive processing (STAP) ** MVDR Estimation Theory See also Probability and Statistics * Estimators ** Biased Estimator ** (Asymptotically) Unbiased Estimator ** Inconsistent Estimator ** (Asymptotically) Consistent Estimator * Methods for evaluating estimators / Optimality criteria ** Minimum-variance unbiased estimator (MVUE) ** Best linear unbiased estimator (BLUE) ** Minimum mean squared error (MMSE) estimator / Bayes least squared error (BLSE) estimator ** James–Stein estimator * Sufficiency, Completeness and Variance Reduction Techniques (VRT) ** Factorization Theorem ** Minimal Sufficient Statistic ** Cramér–Rao bound (CRB) / Cramér–Rao lower bound (CRLB) ** Rao–Blackwell theorem *** Rao–Blackwellization *** Rao–Blackwell estimator ** Exponential family ** Conjugate prior family * Methods for finding estimators ** Method of Moments / Moment Matching ** Maximum Likelihood Estimate (MLE) ** Maximum a posteriori (MAP) ** Pseudolikelihood ** Bayes Estimator / Bayesian Decision Theory *** Conjugate prior *** Exponential family *** Utility Theory ** Minimum-variance unbiased estimator (MVUE) ** Median-unbiased estimator * Frequentist Parameter Estimation ** Frequentist Risk * Bayesian Parameter Estimation / Bayesian Point Estimation / Bayesian Methods ** Posterior Risk / Bayesian Risk ** Bayesian Density Estimation ** Bayes filter / Recursive Bayesian Estimation ** Nonparametric Empirical Bayes (NPEB) ** Parametric Empirical Bayes Point Estimation ** Exponential family ** Conjugate prior * Bayesian Linear Regression * M-estimators ** MLE ** Steepest Descent / Gradient Descent *** Stochastic Gradient Descent (SGD) *** Generalised Normalised Gradient Descent (GNGD) *** Hierarchical Gradient Descent (HGD) *** Normalized Nonlinear Gradient Descent (NNGD) *** Fully Adaptive NNGD (FANNGD) * The Expected Loss Principle * Bayesian Information Theory ** Minimum Message Length (MML) ** Occam's Razor * Model Order Selection / Model Comparison ** Akaike Information Criterion (AIC) ** Bayesian Information Criterion (BIC) ** Minimum Description Length (MDL) ** Akaike Final Prediction Error (FPE) ** Parzen's Criterion Autoregressive Transfer Function (CAT) ** Bayesian Model Selection ** Cross-Validation ** Statistical hypothesis testing (for Multilevel Models / Nested Models only) *** Lagrange multiplier test / Score test *** Likelihood-ratio test *** Wald test System Identification and Subspace Methods See also Machine Learning * Subspace Identification ** N4ASID ** MOESP ** CVA / CCA ** SSARX ** Frequency-domain subspace identification ** Time-domain subspace identification * Linear System Identification ** Linear Grey Box Models / Linear Ordinary Differential Equations ** Linear Black Box Models *** Transfer Function Model *** ARMAX Model *** State-Space Model *** Frequency-Response Model *** Output Error Model *** Box-Jenkins Model * Nonlinear Adaptive Filtering / Nonlinear System Identification ** Blind Deconvolution *** Unsupervised adaptive filter / Blind equalizer / Blind adaptive filter *** Bussgang Algorithm ** Artificial Neural Networks / Adaptive Neural filters *** Feedforward Neural Network *** Recurrent Neural Network ** NARMAX Models ** Volterra Series Models ** Block Structured Models *** Hammerstein systems *** Wiener systems / Parallel cascade nonlinear systems * Subspace Methods / Subspace Learning / Dimensionality Reduction ** Supervised Learning *** Feature Selection ** Unsupervised Learning *** Singular Value Decomposition (SVD) *** Principal Components Analysis (PCA) ** Subspace Tracking *** Grassmannian Rank-One Update Subspace Estimation (GROUSE) *** Parallel Estimation and Tracking by REcursive Least Squares (PETRELS) *** Multiscale Online Union of Subspaces Estimation (MOUSSE) *** Grassmannian Robust Adaptive Subspace Tracking Algorithm (GRASTA) *** Online Supervised Dimensionality Reduction (OSDR) Online Courses Video Lectures * Adaptive filters by Ali H. Sayed Lecture Notes * Advanced Signal Processing by Danilo Mandic * Spectral Estimation and Adaptive Filtering by Danilo Mandic * Stochastic Systems by Florian Herzog * System Identification by Roy Smith * Recursive Estimation by Rafaello D'Andrea Books and Book Chapters * Candy, J. V. (2016). Bayesian signal processing: Classical, modern and particle filtering methods. 2nd Ed. John Wiley & Sons. * Box, G. E., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time series analysis: forecasting and control. John Wiley & Sons. * Kumar, P. R., & Varaiya, P. (2015). Stochastic systems: Estimation, identification, and adaptive control. SIAM. * Haykin , S. (2014). Adaptive filter theory. 5th Ed. Prentice Hall. * Tangirala, A. K. (2014). Principles of System Identification: Theory and Practice. CRC Press. * Goodwin, G. C., & Sin, K. S. (2014). Adaptive filtering prediction and control. Dover Publications. * Åström, K. J., & Wittenmark, B. (2013). Adaptive control. Dover Publications. * Van Trees, H. L. (2013). Detection Estimation and Modulation Theory, Part I: Detection, Estimation, and Filtering Theory. 2nd Ed. John Wiley & Sons. * Farhang-Boroujeny, B. (2013). Adaptive filters: theory and applications. John Wiley & Sons. * Diniz, P. S. (2013). Adaptive filtering. 4th Ed. Springer. * Söderström, T. (2013). Discrete-time stochastic systems: estimation and control. 2nd Ed. Springer Science & Business Media. * Aster, R. C., Borchers, B., & Thurber, C. (2012) Parameter estimation and inverse problems. Academic Press * Sayed, A. H. (2011). Adaptive filters. John Wiley & Sons. * Adali, T., & Haykin, S. (2010). Adaptive signal processing: next generation solutions. John Wiley & Sons. * Hayes, M. H. (2009). Statistical digital signal processing and modeling. John Wiley & Sons. * Mandic, D. P., & Goh, V. S. L. (2009). Complex valued nonlinear adaptive filters: noncircularity, widely linear and neural models (Vol. 59). John Wiley & Sons. * Kulkarni, V. G. (2009). Modeling and analysis of stochastic systems. 2nd Ed. CRC Press. * Porat, B. (2008). Digital processing of random signals: theory and methods. Prentice-Hall. * Vaseghi, S. V. (2008). Advanced digital signal processing and noise reduction. John Wiley & Sons. * Van den Bos, A. (2007). Parameter estimation for scientists and engineers. John Wiley & Sons. * Jazwinski, A. H. (2007). Stochastic processes and filtering theory. Dover Publications. * Poularikas, A. D., & Ramadan, Z. M. (2006). Adaptive filtering primer with MATLAB. CRC Press. * Manolakis, D. G., Ingle, V. K., & Kogon, S. M. (2005). Statistical and adaptive signal processing: spectral estimation, signal modeling, adaptive filtering, and array processing. Artech House. * Anderson, B. D., & Moore, J. B. (2005). Optimal filtering. Dover Publications. * Gray, R. M., & Davisson, L. D. (2004). An introduction to statistical signal processing. Cambridge University Press. * Lehmann, E. L., & Casella, G. (2003). Theory of point estimation. Springer. * Casella, G., & Berger, R. L. (2002). Statistical inference. Cengage Learning. * Cichocki, A., & Amari, S. I. (2002). Adaptive blind signal and image processing: learning algorithms and applications. John Wiley & Sons. * Nelles, O. (2001). Nonlinear system identification: from classical approaches to neural networks and fuzzy models. Springer Science & Business Media. * Moon, T. K., & Stirling, W. C. (2000). Mathematical methods and algorithms for signal processing. Pearson. * Ljung, L. (1999). System identification: theory for the user. PTR Prentice Hall. * West, M. (1999). Bayesian forecasting. John Wiley & Sons. * Bretthorst, G. L. (1998). Bayesian spectrum analysis and parameter estimation. Springer Science & Business Media. * Stoica, P., & Moses, R. L. (1997). Introduction to spectral analysis. Prentice hall. * Van Overschee, P., & De Moor, B. L. (1996). Subspace identification for linear systems: Theory—Implementation—Applications. Springer. * van den Bosch, P. P., & van der Klauw, A. C. (1994). Modeling, identification and simulation of dynamical systems. CRC Press. * Kay, S. M. (1993). Fundamentals of Statistical Signal Processing, Vol I: Estimation Theory. * Therrien, C. W. (1992). Discrete random signals and statistical signal processing. Prentice Hall PTR. * Scharf, L. L. (1991). Statistical signal processing. Addison-Wesley. * Kay, S. M. (1988). Modern spectral estimation. Pearson Education. * Marple, S. L. (1987). Digital Spectral Analysis. Prentice Hall. Scholarly Articles * Geering, H. P., Dondi, G., Herzog, F., & Keel, S. (2011). Stochastic systems. Course script. (link) * Vaidyanathan, P. P. (2007). The theory of linear prediction. Synthesis lectures on signal processing, 2''(1), 1-184. * de Jesús Rubio, J., & Yu, W. (2007). Nonlinear system identification with recurrent neural networks and dead-zone Kalman filter algorithm. ''Neurocomputing, 70(13), 2460-2466. * De Cock, K., De Moor, B., & Leuven, K. U. (2003). Subspace identification methods''.'' Control systems robotics and automation, Vol. 1 of 3, pp. 933-979 * Parlos, A. G., Menon, S. K., & Atiya, A. (2001). An algorithmic approach to adaptive state filtering using recurrent neural networks. IEEE Transactions on Neural Networks, 12(6), 1411-1432. * Griffiths, J. W. R. (1983). Adaptive array processing. A tutorial. Communications, Radar and Signal Processing, IEE Proceedings F, 130(1), 3. * Billings, S. A. (1980). Identification of nonlinear systems: a survey. In IEE Proceedings D-Control Theory and Applications (Vol. 127, No. 6, pp. 272-285). IET. Software * System Identification Toolbox - MATLAB * Statistics and Machine Learning Toolbox - MATLAB * Adaptive Filters (Digital Signal Processing Toolbox) - MATLAB * Signal Modelling (Signal Processing Toolbox) - MATLAB * Spectral Analysis (Signal Processing Toolbox) - MATLAB * Model Selection (Econometrics Toolbox) - MATLAB * Signal Processing (SciPy.Signal) - Python * padasip - Python Adaptive Signal Processing * adaptfilt - Adaptive filtering module for Python * pySPACE - Signal Processing And Classification Environment (SPACE) in Python * pyKalman - Python * spectrum - Spectral Analysis in Python * Scikit.Talkbox - Python * NiTime - Time-Series Analysis for Neuroscience in Python See also * Optimization * Optimal Control * Robust Control * Probabilistic Graphical Models * Computational Modelling Other Resources * Subspace Tracking by Laura Balzano * Course notes on Classical and Bayesian Statistics * AllSignalProcessing.Com - Lessons and MATLAB code * Statistical Signal Processing (CNX.org) - Online free book * Filtering, State Estimation, and Other Forms of Signal Processing - Notebook Category:Signal Processing Category:Probability and Statistics Category:Control Theory Category:Machine Learning